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Abstract 

Objectives: Meta-analyses of biomarkers often present spurious significant results and large effects. We applied sensitivity analyses 
with the use of credibility ceilings to assess whether and how the results of meta-analyses of biomarkers and cancer risk would change. 

Study Design and Setting: We evaluated 98 meta-analyses, 43 (44%) of which had nominally statistically significant results. We 
assumed that any single study cannot give more than a maximum certainty 100 — c% (c, credibility ceiling) that the effect estimate [odds 
ratio (OR)] exceeds 1 (null) or 1.2. 

Results: Nominal statistical significance was maintained for 21 (21%) meta-analyses, for c = 10% and OR > 1, and these proportions 
changed to 7%, 3%, and 6% with ceilings of 20%, 30%, and 40%, respectively. For ceilings for OR > 1.2, the respective proportions were 
37%, 21%, 7%, and 3%. Seven meta-analyses on infectious agents retained statistical significance even with a high ceiling of c = 20% for 
OR > 1.00. Meta-analyses without other hints of bias (large between-study heterogeneity, small-study effects, excess significance) were 
more likely to retain statistical significance than those that had such hints of bias. 

Conclusion: Credibility ceilings may be helpful in meta-analyses of biomarkers to understand the robustness of the results to different 
levels of uncertainty. © 2014 Published by Elsevier Inc. 
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1. Introduction 

Meta-analysis is widely used to synthesize evidence 
from observational studies |1|. A growing literature of 
meta-analysis of observational studies on prognostic and 
other biomarkers suggests that biases may exist that favor 
the reporting of exaggerated associations with significant 
results, often, but not exclusively, in small studies [2—7]. 
Biased results may underlie many of the failures to 
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translate biomarkers into effective clinical tools [8]. 
Because of the large amount of data synthesized in each 
meta-analyses, the confidence intervals (CIs) become 
spuriously narrow and underestimate the biases that may 
exist but often remain unmeasured in the combined non- 
randomized studies. Various methods have been proposed 
to approach this problem, mainly based on Bayesian sta- 
tistics 1 9 1, assuming that different levels of bias affect 
the point estimate or the variance [10—14]. Many of these 
approaches are promising but can be quite convoluted to 
use and may require meticulous, explicit modeling of 
each potential source of bias [15]. Modeling each poten- 
tial source of bias is a very complicated and difficult 
process. There are only a handful of articles in the 
biomedical literature where this has been done, and, even 
then, there is no validation or reassurance that it has led to 
a more accurate result than analyses that were not cor- 
rected for biases. 
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What is new? 
Key findings 

• The application of credibility ceilings in meta- 
analyses of biomarkers and cancer risk led to many 
associations losing their statistical significance. 

What this adds to what was known? 

• Infectious biomarkers were most likely to remain 
statistically significant even with substantial credi- 
bility ceilings being applied. 

• Meta-analyses without other hints of bias were 
more likely to retain statistical significance than 
those that had such hints of bias. 

What is the implication and what should change 
now? 

• Results of meta-analyses of biomarkers should be 
interpreted with adequate caution and may be 
downplayed under the assumption of different 
credibility ceilings for single studies. 



We have previously proposed 1 16] a method for address- 
ing this problem using a simple adjustment of the uncer- 
tainty of the results of the included studies without the 
need to make specific assumptions about the various types 
of implicated bias. The only assumption is that because of 
methodological limitations in observational studies, regard- 
less of sample size and quality, a single study cannot offer 
more than a maximum certainty that an effect is in a partic- 
ular direction (rather than null or even in the other direc- 
tion). The maximum certainty is 100 — c%, where c is 
called the credibility ceiling. 

The aim of the present article was to extend this method 
and apply it not only in one meta-analysis but also across 
many meta-analyses in a specific research field. Moreover, 
we aimed to examine also credibility ceilings that restrict 
the maximum probability of the effect being larger than a 
given nonnull value, rather than larger than the null. This 
approach may fit well in many observational epidemiology 
fields, where biomarkers are involved. Many biomarkers 
may have some small or modest association/prognostic ef- 
fect, whereas few of them have been rigorously, reproduc- 
ibly shown to have large association/prognostic effects. We 
have applied these methods to a previously collected set [5] 
of 98 meta-analyses of biomarker associations with cancer 
risk. The changes in the results are then interpreted accord- 
ing to the adjustment for different credibility ceilings. 
Finally, we also assessed whether meta-analyses that retain 
their statistical significance despite increasing credibility 
ceilings are more likely to be those that do not have other 
hints of bias, for example, small-study effects [17|, excess 



significance [18|, or large between-study heterogeneity 1 19] 
and whether inferences using credibility ceilings may be 
commensurate with inferences obtained by the calculation 
of 95% predictive intervals that accommodate the uncer- 
tainty due to the observed heterogeneity across different 
studies [20 J. 



2. Methods 

2.1. Study identification 

Details on how the eligible meta-analyses were retrieved 
have been reported elsewhere |5|. In brief, we systemati- 
cally searched PubMed from 1966 through December 20, 
2010, to identify meta-analyses of observational studies 
on the association between biomarkers and cancer risk. 
The search strategy used the keywords "cancer or carci- 
noma or tumor or neoplasm or neoplasia or maligna*," 
and they had to appear in the title of the article. When more 
than one meta-analysis on the same research field was 
eligible, the meta-analysis with the largest number of 
studies was retained for the main analysis. 

2.2. Credibility ceiling method 

If we consider the synthesis of n studies, each with an 
effect size y,- with variance v, under a random-effects 
meta-analysis model, the likelihood of each study will be 
Li{fi,di,T^\yi,Vi), a function of the true underlying random 
effect Oi, the common mean effect pi, and the heterogeneity 
variance t^. The likelihood function of the meta-analysis 
model will be L{^l,e,T^\y,v)=]\"^■^Li{p,6i,T^\yi,Vi). 
In the inverse variance meta-analysis framework, 
)u= ^y/W//^ w/ and w, = (v,- + r^)"' |21|. The random- 
effects model inflates the variance of the pooled effect )li 
to incorporate heterogeneity, but the CIs may still be insuf- 
ficiently narrow to include all the factors of uncertainty and 
bias. 

As previously described 1 1 6 1 in the credibility ceiling 
method in a single study, no matter how large and thor- 
oughly conducted, there is at least a probability c that the 
effect will not exceed a certain value X. A credibility ceil- 
ing may be seen as a cumulative likelihood ceiling under 
the situation where for each study we do not consider any 
additional external information about the measure being 
estimated and simply focus on the data from the study 
per se. Limitations in the design and the topic under study 
do not allow c to be reduced further. The value of X can be 
any number, and in many traditional observational associa- 
tions of interest with treatment implications, the main ques- 
tion is whether there is any effect of whatever magnitude. 
For biomarkers, we apply the credibility ceiling method 
for values X = \ (null effect) and X = 1.2 in an odds ratio 
(OR) scale. The reason for this is that in the case of bio- 
markers, many biomarkers have a small or modest prog- 
nostic effect, but far fewer may have a sizable effect. 
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For consistency, we used the symmetry of the ORs and 
reversed the ORs of all studies in all meta-analyses where 
the summary OR was originally less than 1.00 (decreased 
risk of cancer); thus, all summary effects were coined to 
have values >1.00. 

According to the credibility ceiling stipulation, a single 
study cannot give us more than (100 — c)% certainty that 
the effect exceeds each of the plausible values of X. If this 
level of certainty must be exceeded, several studies repli- 
cating an effect greater than X have to be documented. 
Thus, we consider a random variable m, ~ N{yi,Vj) and the 
probability P{u < X\yj > X) that the variable takes values 
less than X is calculated. If this probability is less than c, 
the variance v* is recalculated as v*=mRx[{yi/zc)^, v,] with 
z being the inverse of the cumulative normal distribution. 
The new study-specific likelihood functions with the in- 
flated variances are then synthesized according to inverse 
variance meta-analysis methods, and the variance of the 
summary effect estimate is further inflated as 
var(/x) = l/^ (vt + T^)^\ where v* > v,-. For each credi- 
bility ceiling scenario, we estimate the summary OR ac- 
cording to random-effects calculations (i.e., the point 
estimate of the average effect). The estimated between- 
study heterogeneity would also have to change as it de- 
pends on the study-specific variances. 

Choice of ceiling c values is arbitrary and may be based 
on knowledge of prior evidence or expert opinion. Different 
study designs and research questions should have different 
plausible ceilings. For this reason, we used here a range of 
plausible values so as to explore their impact on the results 
and inferences of the meta-analyses. 

For each credibility ceiling scenario, we estimate the 
summary OR according to random-effects calculations, 
the extent of between-study heterogeneity (fi metric 
[22]), and the likelihood ratio (LR) of the OR exceeding 
vs. not exceeding 1.00. We also calculated the predictive in- 
tervals for each meta-analyses separately, which represent 
the range within which the effect estimate in a new study 
will lie |20|. The prediction interval for the effect size in 
a new study may be approximated by using the formula 

mean±r(if x var(;ti) + t^, where t is the appropriate cen- 

tile point (e.g., 95%) of the t distribution with k — 2 degrees 
of freedom (k is the number of studies). Finally, we 
assessed whether the meta-analyses that retained nominal 
statistical significance (P < 0.05) despite increasing cred- 
ibility ceilings were more likely to be those that had no 
hints of bias based on other tests that we had previously 
applied to these same meta-analyses [5]. These hints of po- 
tential bias included small-study effects 117], excess signif- 
icance |18|, and large between-study heterogeneity [19]. 
Small-study effects, meaning whether smaller studies give 
larger estimates of effect size compared with larger studies, 
were evaluated with the Egger regression asymmetry test 
[23]. Excess statistical significance was evaluated using 
the excess significance test [18,24], which evaluates 



whether the observed number of studies with nominally sta- 
tistically significant results is too large based on the power 
that these studies have to detect plausible effects at 
a = 0.05. Large between-study heterogeneity was defined 
as point estimates greater than 50% [19]. 

All calculations were performed in Stata 10 (StataCorp, 
2007, Stata Statistical Software: Release 10, StataCorp LP, 
College Station, TX, USA). 



3. Results 

3.1. Description of the included meta-analyses 

The selection of the included studies has been described 
in Tsilidis et al. [5]. In brief, 65 articles were reviewed in 
full text, and 37 articles were finally selected [25— 61|, 
which provided data on 98 meta-analyses (including 847 
studies) in six broad fields of biomarkers and cancer risk 
[insulinlike growth factor (IGF)/insulin system (n = 21 
comparisons), sex hormones {n = 13 comparisons), diet 
(n = 31 comparisons), inflammation (n = 3 comparisons), 
infectious agents (n = 22 comparisons), and environment 
(m = 8 comparisons)] (Table 1). The predictive intervals 
did not include the null value in 10 meta-analyses (two in 
IGF/insulin system, one in sex hormones, one in diet, and 
six on infectious agents). 

3.2. Comparison of results with ceiling 0% and 10% for 
OR >1.00 

As shown in Table 1, without applying correction for 
any ceiling, 43 meta-analyses (44%) had a statistically sig- 
nificant finding (34 showed increased risk of cancer). The 
proportion of statistically significant findings in each field 
was 86% on infectious agents, 67% on inflammation, 
52% on IGF/insulin, 31% on sex hormones, 19% on diet, 
and 13% on environmental factors. The point estimate 
was >0 in 69 meta-analyses (70%). 

As an illustration (Table 1), with an application of a 
typical ceiling of 10% for OR > 1.00, only approximately 
half of the meta-analyses retained their statistical signifi- 
cance (21%), ranging from 64% on infectious agents to 
0% on environmental factors and inflammation. The pro- 
portion of studies with f' point estimate >0 was also 
reduced to about half (36%). 

3.3. Consideration of different ceilings 

Table 2 summarizes the changes in the effect estimates, 
f', and LR for all scenarios of different credibility ceilings 
for all meta-analyses. Detailed information for changes in 
the summary OR, fi, and LR for credibility ceilings 
0.05-0.40 for OR >1 and for OR >1.2 is shown in 
Supplementary Figures 1—6, and Supplementary Table 1 
at www.jclinepi.com summarizes data for each field 
(different types of risk factors). 
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Table 1. Analytical description of the 98 selected meta-analyses and predictive intervals with random effects (95% CI), / , and LR for credibility 
ceilings 0% and 10% under the scenario of the effect to be at the opposite direction 



Meta-analysis 
(Ref) 


Comparison 


Studies 


Cases/controls 


Ceiling 0% 




Ceiling; 10% 




Predictive 


Random effects 


(%) 


LR 


Random effects 


P (%) 


LR 


intervals 


IGF/insulin 






















[32] 


IGF-1 and CRC 


11 


2,862/4,966 


1.07 (1.01, 1.14) 


0 


79 


1.07 (1.00, 1.14) 


0 


43 


1.00-1.16 


[33] 


IGFBP-3 and CRC 


7 


1,106/2,395 


0.98 (0.64, 1.51) 


60 


1.15 


0.97 (0.64, 1.45) 


50 


0.8 


0.27-3.50 


[33] 


IGF-2 and CRC 


3 


384/1,301 


1.95 (1.26, 3.00) 


0 


792 


1.84 (1.03, 3.30) 


0 


48 


0.12-32.2 


[34] 


IGF-1 and 


11 


1,937/4,096 


1.21 (0.98, 1.49) 


29 


26 


1.12 (0.94, 1.33) 


0 


8 


0.74-1.97 



[34] 



[34] 



[34] 



[35] 
[35] 

[36] 
[36] 
[36] 
[36] 

[36] 
[36] 
[37] 
[37] 
[37] 

[37] 

[37] 
[37] 

Sex hormones 
[38] 
[38] 
[38] 
[38] 
[38] 
[38] 
[38] 
[38] 
[38] 
[39] 



[40] 
[40] 

[40] 

Diet 
[41] 

[42] 

[43] 

[43] 

[44] 
[44] 



premenopausal 

BrCA 
IGF-1 and 

postmenopausal 

BrCA 
IGFBP-3 and 

premenopausal 

BrCA 
IGFBP-3 and 

postmenopausal 

BrCA 
IGF-1 and lung CA 
IGFBP-3 and lung 

CA 

IGF-1 and PrCA 
IGF-2 and PrCA 
IGFBP-3 and PrCA 
IGF-l/BP-3 and 

PrCA 
IGFBP-1 and PrCA 
IGFBP-2 and PrCA 
C-peptide and CRC 
C-peptide and BrCA 
C-peptide and 

endometrial CA 
C-peptide and 

pancreas CA 
Glucose and CRC 
Glucose and 

pancreas CA 

T and PrCA 
Free T and PrCA 
DHT and PrCA 
A-diol G and PrCA 
DHES-S and PrCA 
D4 and PrCA 
E2 and PrCA 
Free E2 and PrCA 
SHBG and PrCA 
E2 and 

postmenopausal 

BrCA 
20HE1 and PrCA 
16a-0HEl and 

PrCA 
20HEl/16a-0HEl 

and PrCA 

25(0H) vitamin D 

and colon CA 
25(0H) vitamin D 

and PrCA 
25(0H) vitamin D 

and BrCA 
la,25(0H)2 

vitamin D and 

BrCA 
Carotenoids and 

lung CA 
B-carotene and 

lung CA 



15 



11 



15 



6 
6 

42 
10 
29 
11 



11 
5 



7 
11 
7 
3 



2,853/5,332 1.30 (1.13, 1.49) 



1,915/4,012 0.99 (0.83, 1.19) 



2,816/5,196 1.22(1.01,1.49) 31 



> 1,000 



1.20 (1.01, 1.40) 



1.15 0.98 (0.81, 1.18) 



1,043/11,472 
1,043/11,472 

7,481/11,866 

923/1,874 
6,676/10,484 
3,545/6,132 



0.98 (0.68, 1.41) 
0.96 (0.59, 1.56) 



41 
67 



45 



1.15 
1.3 



1.08 (0.91, 1.29) 



1.02 (0.71, 1.46) 
0.99 (0.60, 1.62) 



41 
57 



209/483 



1,741/1.38 m 
298/1.33 m 



1.70 (1.11, 2.61) 

1.28 (1.06, 1.54) 
1.98 (1.67, 2.35) 



47 
0 



136 

190 
> 1,000 



1.64 (0.95, 2.83) 

1.12 (0.99, 1.27) 
1.80 (1.17, 2.77) 



122/414 
122/414 



0.76 (0.45, 1.28) 
1.82 (1.08, 3.05) 



122/414 0.52 (0.31, 0.89) 



0 



5.8 1.13 (0.78, 2.24) 
84 1.86(0.94,3.67) 



120 



1.96 (0.94, 3.67) 



0 



984/1,960 0.78(0.53,1.13) 46 

3,124/4,682 1.03(0.95,1.11) 26 

5,489/5,841 0.55 (0.38, 0.80) 86 

1,802/1,825 0.99(0.68,1.44) 47 



4 538/7,265 0.70(0.44,1.11) 45 
10 2,846/34,783 0.84 (0.66, 1.07) 41 



9.6 1.14 (0.86, 1.52) 1 

3.1 1.02 (0.95, 1.09) 13 

> 1,000 1.23 (0.99, 1.53) 4 

1.1 1.00 (0.69, 1.44) 40 

15 1.26 (0.88, 1.79) 0 

11 1.13 (0.87, 1.47) 31 



57 



1.12-1.51 



0.7 0.80-1.22 



0.75-2.00 



1.1 
0.9 



0.38-2.53 
0.21-4.35 



1. 


,21 (1.07, 


1.36) 


88 


880 


1.02 (0.95, 


1.09) 


46 


2 


0.58-2.51 


1. 


17 (0.93, 


1.47) 


77 


10 


1.07 (0.87, 


1.31) 


47 


3 


0.55-2.50 


0. 


,88 (0.79, 


0.98) 


81 


118 


1.01 (0.95, 


1.06) 


24 


1 


0.52-1.47 


1. 


10 (0.97, 


1.24) 


80 


15 


1.01 (0.94, 


1.09) 


30 


1.5 


0.72-1.69 



3 


617/936 


1 


.20 (0.65, 


2.22) 


92 


2.5 


0.84 (0.63, 


1 


.12) 


50 


0.1 


5 


844/1,826 


1 


.18 (0.90, 


1.54) 


78 


7.5 


0.97 (0.85, 


1 


.10) 


0 


0.5 


12 


1,309/4,233 


1 


.51 (1.14, 


1.99) 


54 


501 


1.20 (0.96, 


1 


.51) 


10 


17 


11 


1,403/2,114 


1 


.35 (1.01, 


1.81) 


64 


43 


1.07 (0.90, 


1 


.28) 


0 


3 


4 


388/474 


1 


.18 (0.57, 


2.43) 


69 


2 


0.89 (0.58, 


1 


.35) 


0 


0.41 



0, 


.00- 


-2,521 


0, 


.46- 


-3.04 


0, 


.66- 


-3.45 


0, 


.53- 


-3.41 


0, 


.05- 


-25.4 



24 



30 
266 



Inestimable 



0.79- 
1.50- 



2.07 
2.61 



17 


3,886/6,438 


0.98 (0.87, 


1 


.10) 


0 


2 


0 


.98 (0.86, 


1 


.11) 


0 


0.5 


0.86- 


-1 


.11 


14 


3,550/5,815 


1.12 (0.98, 


1 


.27) 


0 


21 


1 


.10 (0.97, 


1 


.26) 


0 


15 


0.97- 


-1 


.28 


7 


1,010/1,445 


0.88 (0.69, 


1 


.11) 


0 


6 


1 


.14 (0.90, 


1 


.45) 


0 


6 


0.64- 


-1 


.20 


8 


2,453/3,035 


1.15 (0.95, 


1 


.38) 


24 


12 


1 


.08 (0.92, 


1 


.28) 


0 


4.3 


0.78- 


-1 


.69 


7 


1,154/1,870 


1.29 (0.99, 


1 


.68) 


17 


30 


1 


.17 (0.92, 


1 


.47) 


0 


9.4 


0.76- 


-2, 


.17 


6 


1,781/2,430 


1.02 (0.85, 


1 


.21) 


0 


1.3 


1 


.02 (0.85, 


1 


.21) 


0 


1.3 


0.79- 


-1 


.31 


9 


2,186/3,039 


0.92 (0.78, 


1 


.09) 


0 


5.1 


1 


.09 (0.92, 


1 


.28) 


0 


5 


0.75- 


-1 


.12 


8 


2,043/2,735 


0.97 (0.82, 


1 


.16) 


0 


1.6 


1 


.00 (0.83, 


1 


.19) 


0 


1 


0.79- 


-1 


.21 


15 


3,704/5,998 


0.86 (0.76, 


0 


.97) 


0 


149 


1 


.14 (1.00, 


1 


.30) 


0 


41 


0.75- 


-0, 


.98 


9 


656/1,709 


1.26 (1.07, 


1, 


.49) 


42 


416 


1 


.14 (0.97, 


1, 


.32) 


0 


17 


0.83- 


-1, 


.92 



6 
26 

27 

5 
2 
33 
1 



Inestimable 
Inestimable 

Inestimable 

0.28-2.11 
0.87-1.22 
0.16-1.92 
0.02-42.5 

0.13-3.78 
0.46-1.54 



(Continued) 
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Table 1. Continued 



Meta-analysis 
(Ref) 


Comparison 


Studies 


Cases/controls 






Celling 0% 






Celling 10% 




Predictive 
Intervals 


Random effects 


F (%) 


LR 


Random effects 


F (%) 


LR 


[44] 


A-carotene and 


5 


835/4,783 


0.88 


(0.59, 


1.33) 


53 


2.5 


1.00 (0.74, 


1.33) 


1 


1 


0.25-3.13 




lung OA 


























[44] 


B-cryptoxanthin 


5 


835/4,783 


0.82 


(0.40, 


1.59) 


75 


2.3 


1.15 (0.55, 


2.42) 


70 


2 


0.07-9.88 




and lung CA 


























[44] 


Lutein/zeaxanthin 


4 


559/4,507 


0.95 


(0.57, 


1.35) 


11 


1.5 


1.00 (0.70, 


1.42) 


0 


1 


0.37-2.45 




and lung CA 


























[44] 


Lycopene and lung 


4 


727/4,567 


0.71 


(0.51, 


0.99) 


0 


48 


1.32 (0.91, 


1.90) 


0 


13 


0.34-1.45 


[45] 


CA 

Selenium and lung 
CA 


5 


1,029/1,658 


0.77 


(0.55, 


1.08) 


42 


14 


1.18 (0.85, 


1.52) 


20 


6 


0.33-1.84 


[46] 


SFA and BrCA 


6 


1,085/1,485 


1.05 


(0.79, 


1.39) 


0 


1.7 


1.02 (0.75, 


1.36) 


0 


1 


0.70-1.57 


[46] 


Stearic acid and 


7 


1,295/1,507 


0.93 


(0.59, 


1.26) 


14 


2.1 


1.03 (0.77, 


1.37) 


0 


1 


0.53-1.64 




BrCA 


























[46] 


Palmitic acid and 


7 


1,295/1,507 


1.05 


(0.59, 


1.58) 


59 


1.4 


1.00 (0.57, 


1.50) 


51 


1 


0.31-3.52 




BrCA 


























[46] 


MUFA and BrCA 


5 


931/1,360 


1.44 


(0.82, 


2.53) 


57 


8.8 


1.15 (0.79, 


1.73) 


22 


3 


0.22-9.54 


[46] 


Palmitoleic acid 


2 


267/531 


1.25 


(0.41, 


3.89) 


81 


1.9 


1.12 (0.37, 


3.35) 


51 


1 


Inestimable 




and BrCA 


























[46] 


Oleic acid and 


9 


1,740/1,983 


0.99 


(0.70, 


1.38) 


70 


1.14 


0.92 (0.58, 


1.24) 


39 


0.4 


0.34-2.82 




BrCA 


























[46] 


n-6 PUPA and BrCA 


7 


1,206/1,461 


0.75 


(0.53, 


1.06) 


15 


18 


1.24 (0.88, 


1.75) 


1 


8 


0.39-1.45 


[46] 


Linoleic acid and 


8 


1,449/1,632 


0.85 


(0.57, 


1.26) 


50 


3.7 


1.05 (0.73, 


1.52) 


42 


1 


0.27-2.73 




BrCA 


























[46] 


Arachidonic acid 


5 


975/1,251 


0.89 


(0.55, 


1.22) 


0 


3.2 


1.07 (0.77, 


1.50) 


0 


2 


0.54-1.48 




and BrCA 


























[46] 


n-3 PUFAand BrCA 


8 


1,360/1,586 


0.79 


(0.55, 


1.11) 


37 


10 


1.17 (0.85, 


1.52) 


19 


5 


0.34-1.84 


[46] 


a-Linolenic acid 


8 


1,586/1,858 


0.80 


(0.59, 


1.08) 


39 


12.3 


1.11 (0.87, 


1.42) 


0 


4 


0.38-1.70 




and BrCA 


























[46] 


Eicosapentaenoic 


5 


931/1,360 


0.91 


(0.87, 


0.95) 


0 


> 1,000 


1.13 (0.99, 


1.30) 


0 


29 


0.84-0.97 




acid and BrCA 


























[46] 


Docosahexaenoic 


7 


1,463/1,799 


0.73 


(0.53, 


1.02) 


35 


30 


1.17 (0.88, 


1.55) 


0 


6 


0.33-1.64 




acid and BrCA 


























[47] 


Enterolactone and 


12 


3,088/4,622 


0.79 


(0.51, 


1.02) 


71 


29 


1.00 (0.83, 


1.20) 


18 


1 


0.33-1.87 




BrCA 


























[48] 


a-Linolenic acid 


6 


1,091/1,270 


1.54 


(1.15, 


2.06) 


16 


631 


1.35 (1.01, 


1.82) 


0 


45 


0.87-2.75 




and PrCA 


























[49] 


Folate and BrCA 


5 


1,239/2,345 


0.67 


(0.45, 


1.00) 


41 


41 


1.30 (0.95, 


1.79) 


0 


18 


0.25-1.94 


[50] 


Folate and PrCA 


7 


2,958/6,962 


1.11 


(0.95, 


1.28) 


39 


11 


1.02 (0.95, 


1.09) 


0 


3 


0.79-1.55 


[50] 


Vitamin B12 and 


5 


2,906/6,495 


1.10 


(1.01, 


1.19) 


45 


89 


1.05 (1.00, 


1.13) 


0 


39 


0.89-1.35 




PrCA 


























[50] 


Total homocysteine 


4 


2,028/4,987 


0.91 


(0.70, 


1.19) 


14 


3 


1.05 (0.83, 


1.33) 


0 


2 


0.43-1.91 




and PrCA 


























[51] 


Vitamin B6 and 


4 


883/1,424 


0.52 


(0.38, 


0.71) 


0 


> 1,000 


1.85 (1.15, 


2.98) 


0 


173 


0.25-1.03 




CRC 


























Inflammation 




























[52] 


C-reactive protein 


8 


1,159/37,985 


1.12 


(1.01, 


1.25) 


51 


73 


1.07 (0.98, 


1.17) 


22 


13 


0.85-1.49 




and CRC 


























[53] 


C-reactive protein 


14 


4,438/70,107 


1.10 


(1.02, 


1.18) 


73 


227 


1.03 (0.98, 


1.08) 


18 


5 


0.85-1.42 




and CA 


























[53] 


lnterleukin-6 and 
CA 


4 


868/5,917 


1.01 


(0.90, 


1.12) 


21 


1.2 


1.01 (0.90, 


1.11) 


20 


1 


0.73-1.38 


Infection 




























[26] 


H pylori and early 


15 


2,722/13,975 


3.38 


(2.15, 


5.32) 


83 


> 1,000 


1.71 (1.20, 


2.45) 


4 


651 


0.51-18.5 




gastric CA 


























[54] 


H pylori and gastric 


15 


2,284/2,770 


2.29 


(1.71, 


3.05) 


76 


> 1,000 


1.45 (1.15, 


1.82) 


0 


> 1,000 


0.77-5.82 


[54] 


CA 

cagA and gastric CA 


10 


1,707/2,124 


2.87 


(1.95, 


4.22) 


85 


> 1,000 


1.74 (1.27, 


2.38) 


0 


> 1,000 


0.73-11.3 


[55] 


H pylori and CRC 


14 


1,709/1,872 


1.49 


(1.15, 


1.90) 


58 


> 1,000 


1.15 (0.97, 


1.39) 


0 


18 


0.57-3.30 


[55] 


H pylori and 


3 


108/249 


2.02 


(1.27, 


3.23) 


0 


646 


1.87 (1.04, 


3.35) 


0 


54 


0.10-41.8 




laryngeal CA 


























[56] 


H pylori and 


13 


840/2,890 


0.57 


(0.47, 


0.59) 


15 


> 1,000 


1.45 (1.15, 


1.84) 


0 


> 1,000 


0.39-0.82 




esophageal 




























adeno-CA 


























[56] 


H pylori and ESCC 


9 


921/2,743 


1.10 


(0.78, 


1.55) 


73 


2.4 


1.02 (0.73, 


1.43) 


55 


1 


0.35-3.35 


[56] 


cagA and 


5 


275/1,197 


0.41 


(0.28, 


0.52) 


17 


> 1,000 


1.84 (1.13, 


3.02) 


0 


134 


0.17-1.02 




esophageal 




























adeno-CA 


























[56] 


cagA and ESCC 


4 


483/1,844 


1.01 


(0.79, 


1.27) 


0 


1.0 


1.00 (0.79, 


1.27) 


0 


1 


0.50-1.69 


[28] 


H pylori and lung 


4 


199/231 


3.24 


(1.11, 


9.41) 


79 


64 


1.47 (0.85, 


2.50) 


0 


12 


0.03-371 



{Continued) 
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Ceiling 0% 



Ceiling 10% 



Predictive 



(Ref) 


Comparison 


Studies 


Cases/controls 


Random effects 


(%) 


LR 


Random effects 


(%) 


LR 


intervals 


[57] 


HPV (DNA) and 


13 


478/179 


2.30 


(1.33, 


4.00) 


5 


653 


1.81 


(0.97, 


3.37) 


0 


31 


1.00-1.31 




bladder CA 




























[57] 


HPV (no DNA) and 


3 


175/203 


2.98 


(1.65, 


5.40) 


0 


> 1,000 


3.45 


(1.06, 


11.2) 


0 


50 


0.06-140 




bladder CA 




























[30] 


HPV and oral CA 


8 


1,541/2,335 


1.99 


(1.17, 


3.38) 


62 


178 


1.41 


(0.94, 


2.13) 


0 


19 


0.46-8.55 


[30] 


HPV and larynx CA 


8 


412/721 


2.01 


(0.96, 


4.22) 


50 


30 


1.49 


(0.78, 


2.82) 


15 


8 


0.29-13.9 


[30] 


HPV and 


5 


383/1,815 


4.31 


(2.07, 


8.95) 


55 


> 1,000 


3.05 


(1.27, 


7.30) 


0 


151 


0.49-38.2 




oropharynx CA 




























[30] 


HPV and tonsil CA 


8 


217/153 


15.1 


(5.78, 


33.4) 


0 


> 1,000 


13.3 


(3.04, 


58.5) 


0 


> 1,000 


5.56-40.7 


[31] 


HPV and PrCA 


9 


2,242/2,522 


1.52 


(1.12, 


2.05) 


35 


258 


1.25 


(1.00, 


1.57) 


0 


39 


0.74-3.09 


[58] 


HPV and cervical 
CA 


12 


589/3,058 


8.08 


(5.04, 


10.8) 


27 


> 1,000 


5.93 


(2.84, 


16.9) 


0 


> 1,000 


4.22-15.5 


[29] 


HBV (HCV-) and 


28 


2,881/5,318 


21.9 


(14.9, 


32.3) 


85 


> 1,000 


3.71 


(2.16, 


5.36) 


0 


> 1,000 


3.13-153 




liver CA 




























[29] 


HCV (HBV-) and 


25 


2,079/5,515 


20.3 


(12.2, 


33.7) 


86 


> 1,000 


7.04 


(3.64, 


13.6) 


0 


> 1,000 


1.68-245 




liver CA 




























[29] 


HBV + HCV and 


9 


567/1,870 


51.2 


(27.0, 


139) 


37 


> 1,000 


28.2 


(4.79, 


166) 


0 


> 1,000 


8.06-455 




liver CA 




























[51] 


T vaginalis and 


2 


542/55,122 


1.88 


(1.29, 


2.74) 


0 


> 1,000 


1.70 


(0.94, 


3.06) 


0 


24 


Inestimable 




cervical CA 




























Environment 






























[59] 


Dieldrin and BrCA 


5 


1,550/1,573 


1.15 


(0.77, 


1.69) 


43 


3 


1.02 


(0.73, 


1.41) 


0 


1 


0.37-3.52 


[59] 


Oxychlordane and 


5 


1,311/1,407 


0.77 


(0.51, 


1.14) 


51 


9 


1.18 


(0.80, 


1.73) 


25 


4 


0.22-2.51 




BrCA 




























[59] 


Cis-nonachlor and 


3 


692/595 


1.09 


(0.72, 


1.64) 


0 


1.9 


1.09 


(0.72, 


1.64) 


0 


2 


0.08-15.7 




BrCA 




























[59] 


Trans-nonachlor 


5 


1,485/1,752 


0.86 


(0.68, 


1.07) 


0 


10.2 


1.14 


(0.90, 


1.45) 


0 


5 


0.62-1.18 




and BrCA 




























[50] 


DDT and BrCA 


24 


5,222/5,147 


0.97 


(0.85, 


1.11) 


17 


1.9 


1.01 


(0.89, 


1.14) 


5 


1 


0.72-1.32 


[25] 


DNA adducts and 


8 


509/407 


3.76 


(1.75, 


8.05) 


94 


> 1,000 


1.44 


(0.86, 


2.40) 


15 


2 


0.28-50.0 




CA (current 






























smokers) 




























[25] 


DNA adducts and 


7 


295/337 


0.94 


(0.71, 


1.25) 


0 


2 


1.07 


(0.80, 


1.41) 


0 


0.5 


0.65-1.35 




CA (for smokers) 




























[25] 


DNA adducts and 


9 


224/340 


1.64 


(0.72, 


3.77) 


79 


7.3 


0.91 


(0.56, 


1.46) 


33 


0.48 


0.12-22.43 




CA (never 






























smokers) 





























Abbreviations: CI, confidence interval; LR, likelihood ratio; IGF, insulinlike growth factor; CRC, colorectal cancer; IGFBP, insulinlike growth 
factor— binding protein; BrCA, breast cancer; CA, cancer; PrCA, prostate cancer; ESCC, esophageal squamous cell carcinoma; T, testosterone; 
DHT, dihydrotestosterone; A-diol G, androstanediol glucuronide; DHES-S, dehydroepiandrosterone sulfate; D4, androstenedione; E2, estradiol; 
SHBG, sex hormone— binding globulin; El, estrone; SFA, total saturated fatty acids; MUFA, monounsaturated fatty acids; PUFA, polyunsaturated 
fatty acids; H pylori, Helicobacter pylori; HPV, human papillomavirus; HBV, hepatitis B virus; HCV, hepatitis C virus; T vaginalis, Trichomonas vag- 
inalis; DDT, dichlorodiphenyltrichloroethane. 

Random effects refers to summary odds ratio (95% CI) using the random-effects model. 



With a ceiling of 20% for OR > 1.0, only seven meta- 
analyses (7%) preserved statistical significance. Fifteen 
percent of the meta-analyses still had a nonzero point esti- 
mate of f. With a ceiling of 30% for OR > 1.0, three meta- 
analyses (3%) retained statistical significance, and nine 



(9%) had a nonzero point estimate of f. For a ceiling of 
40% for OR > 1 .0, six meta-analyses (6%) appear to have 
a statistically significant summary result. This spurious in- 
crease in the number of statistically significant meta- 
analyses occurred because with the reduction in the 



Table 2. Changes in the proportion of studies with effect estimates >X, nominal statistical significance, and nonzero F for all scenarios of the 
magnitude of the effect for all meta-analyses inclusively 



All meta-analyses 


Celling 0% 


Ceiling 5% 


Ceiling 10% 


Ceiling 20% 


Ceiling 30% 


Ceiling 40% 


X= 1, n (%) 














No. of meta-analyses with effect size > 1.0 


98 (100) 


92 (94) 


83 (85) 


69 (70) 


48 (49) 


29 (30) 


No. of meta-analyses with nominal statistical significance 


43 (44) 


34 (35) 


21 (21) 


7 (7) 


3 (3) 


6 (6) 


No. of meta-analyses with nonzero 


69 (70) 


57 (58) 


35 (36) 


15 (15) 


9 (9) 


1 (1) 


X= 1.2, n (%) 














No. of meta-analyses with effect size > 1.2 


54 (55) 


50 (51) 


42 (43) 


33 (34) 


25 (25) 


20 (20) 


No. of meta-analyses with nominal statistical significance 


43 (44) 


40 (41) 


36 (37) 


21 (21) 


7 (7) 


3 (3) 


No. of meta-analyses with nonzero F 


69 (70) 


61 (62) 


52 (53) 


28 (29) 


21 (21) 


13 (13) 
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summary effect estimates, some of the meta-analyses 
gained statistical significance but on the opposite direction 
of the initial estimate; no meta-analysis retained statistical 
significance with an effect in the original direction with 
such a ceiling. Only one meta-analysis had a nonzero point 
estimate of I^. 

Considering various ceilings for OR >1.2, half of the 
meta-analyses (23%) lost statistical significance for a ceiling 
of 20%. Seven meta-analyses kept statistical significance for 
a ceiling of 30%, whereas only three meta-analyses survived 
the 40% ceiling. Between-study heterogeneity dropped 
gradually with higher ceilings (Table 2). 

3.4. Meta-analyses with statistically significant results 
despite large ceilings 

Table 3 lists the change in the summary results for 
different ceilings for each of the seven meta-analyses that 
had nominally statistically significant results when ceilings 
of 20% or higher are applied for OR > 1.0, and Table 4 lists 
the changes in the effect estimates for each of the 21 meta- 
analyses that had nominally statistically significant results 
with the application of ceilings of 20% or higher for 
OR >1.2. 

All seven meta-analyses that retained statistical signifi- 
cance with c = 20% for OR > 1 were on infectious agents 
[cagA strains of Helicobacter pylori and gastric cancer, 
H pylori and esophageal adenocarcinoma, human papillo- 
mavirus (HPV) and tonsil/cervical cancer, and hepatitis B 
virus and/or hepatitis C virus (HCV) and liver cancer]. 
The meta-analysis on HPV and cervical cancer remained 
statistically significant even with 30% ceiling, whereas 
none survived a 40% ceiling. 

The seven meta-analyses that retained statistical signifi- 
cance with c = 30% for OR > 1 .2 included three meta- 
analyses of infectious agents (cagA strains of H pylori 
and gastric cancer, H pylori and esophageal cancer, and 
HCV and liver cancer) but also four other meta-analyses 
[IGF-I and colorectal cancer, eicosapentaenoic acid and 
breast cancer, vitamin B12 and prostate cancer, and C- 
reactive protein (CRP) and cancer]. Two meta-analyses 



retained statistical significance with a ceiling of 40% (eico- 
sapentaenoic acid and breast cancer and vitamin B12 and 
prostate cancer), whereas another one (IGFBP-3 and pros- 
tate cancer) gained statistical significance but with the ef- 
fect being in the opposite direction. 

3.5. Comparison against other bias-testing tools 

Of the 24 statistically significant meta-analyses that had 
been previously identified 1 5 1 to have no signs of bias (no 
small-study effects, no large heterogeneity, and no excess 
of significant studies), the number that remained significant 
with c = 5%, 10%, 20%, and 30% for OR > 1.0 was 23 
(96%), 15 (63%), 5 (21%), and 1 (4%), respectively. Of 
the 19 statistically significant meta-analyses that had been 
previously identified to have at least one such hint of poten- 
tial bias, the respective numbers were 11 (58%), 6 (32%), 2 
(11%), and 0 (0%). For OR >1.2, the number of meta- 
analyses that retained statistical significance for c = 5%, 
10%, 20%, 30% was 24 (100%), 22 (92%), 13 (54%), 
and 4 (17%) in the group without hints of bias and 16 
(84%), 14 (74%), 8 (42%), and 3 (16%), respectively, in 
those with hints of bias. Thus, there was some suggestion 
that statistical significance was more likely to be retained 
with various ceilings in meta-analyses that had no hints 
of bias (e.g., P = 0.048 for c = 10% for OR > 1.0). The 
difference was more prominent for ceilings related to 
OR > 1.0 rather than OR > 1.2. 

3.6. Comparison against predictive intervals 

Of the 10 meta-analyses with 95% predictive intervals 
excluding the null, the number that remained significant 
with c = 5%, 10%, 20%, and 30% for OR >1.0 was 10 
(100%), 9 (90%), 6 (60%), and 1 (1%), respectively. Of 
the 33 meta-analyses with 95% CIs excluding the null, 
but with 95% predictive intervals including the null, the 
respective numbers were 24 (73%), 12 (36%), 1 (3%), 
and 1 (3%). For OR >1.2, the number of meta-analyses 
that retained statistical significance for c = 5%, 10%, 
20%, and 30% was 10 (100%), 10 (100%), 10 (100%), 



Table 3. Changes in summary effect estimates under tlie random-effects model for ceilings 0-40% for summary OR > 1 for tine seven studies that 
had a statistically significant summary effect for large ceilings 

Ceiling 



Comparison Ceiling 0% Ceiling 5% Ceiling 10% Ceiling 20% Ceiling 30% 40% 

cagA and gastric OA 2.87(1.95-4.22) 1.74(1.36-1.22) 1.74(1.27-2.38) 1.74(1.08-2.8) NS NS 

Hpy/or; and esophageal 1.77(1.46-2.14)" 1.52(1.23-1.89) 1.46(1.15-1.84) 1.35(1.02-1.79) NS NS 
adeno-CA 

HPV and tonsil OA 15.1(6.78-33.4) 14.46(4.36-13.9) 13.34(3.04-58.5) 11.6(1.44-93.1) NS NS 

HPV and cervical CA 8.08(6.04-10.8) 6.93(3.46-13.88) 6.93(2.84-16.9) 6.93(1.78-26.9) 6.96(1.41-34.3) NS 

HBV(HCV-) and liver OA 21.9(14.9-32.3) 9.38(5.07-17.36) 3.71(2.17-6.36) 2.69(1.37-5.3) NS NS 

HCV (HBV-) and liver CA 20.3(12.2-33.7) 20.2(12.2-33.7) 7.04(3.64-13.6) 6.97(2.58-18.8) NS NS 

HBV + HCV and liver CA 61.2(27.0-139) 61.2(27.0-139.13) 28.2(4.8-165.7) 28.2(1.9-418) NS NS 



Abbreviations: OR, odds ratio; CA, cancer; NS, not specified; H pylori, Helicobacter pylori; HPV, human papillomavirus; HBV, hepatitis B virus; 
HCV, hepatitis C virus. 

" Effect inversed so that it is > 1.00. 
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Table 4. Changes in summary effect estimates under tlie random-effects model for ceilings 0—40% for summary OR > 1.2 for tlie 21 studies tliat 
had a statistically significant summary effect for large ceilings 

Comparison Ceiling 0% Ceiling 5% Ceiling 10% Ceiling 20% Ceiling 30% Ceiling 40% 

IGF-1 and CRC 1.07(1.01-1.14) 1.07(1.01-1.14) 1.07(1.01-1.14) 1.07(1.01-1.14) 1.07(1.01-1.14) NS 

IGF-1 and 1.30(1.13-1.49) 1.29(1.12-1.48) 1.26(1.09-1.45) 1.19(1.02-1.40) NS NS 

postmenopausal 

BrCA 

Glucose and CRC 1.28(1.05-1.54) 1.21(1.03-1.43) 1.15(1.02-1.30) 1.12(1.01-1.25) NS NS 

Glucose and 1.98(1.67-2.35) 1.81(1.38-2.38) 1.78(1.31-2.44) 1.77(1.14-2.75} NS NS 
pancreas CA 

SHBGandPrCA 1.17(1.03-1.32)^ 1.17(1.03-1.32) 1.17(1.03-1.32) 1.15(1.01-1.30) NS NS 

E2 and 1.25(1.07-1.49) 1.25(1.07-1.48) 1.22(1.04-1.43) 1.17(1.02-1.34) NS NS 

postmenopausal 

BrCA 

25(0H) vitamin D 1.82(1.25-2.63)" 1.45(1.11-1.91) 1.34(1.05-1.70) 1.22(1.01-1.47) NS NS 
and BrCA 

Eicosapentaenoic 1.10(1.05-1.15)" 1.10(1.05-1.15) 1.10(1.06-1.15) 1.10(1.05-1.15) 1.10(1.05-1.15) 1.10(1.05-1.15) 
acid and BrCA 

Vitamin B12 and 1.10(1.01-1.19) 1.09(1.01-1.17) 1.07(1.02-1.13) 1.07(1.01-1.13) 1.05(1.01-1.13) 1.05(1.01-1.12) 
PrCA 

Vitamin B6 and 1.91(1.40-2.61)" 1.91(1.40-2.61) 1.87(1.29-2.71) 1.82(1.10-3.01) NS NS 
CRC 

C-reactive protein 1.12(1.01-1.25) 1.12(1.01-1.25) 1.12(1.01-1.25) 1.12(1.01-1.23) NS NS 
and CRC 

C-reactive protein 1.10(1.02-1.18) 1.10(1.02-1.18) 1.09(1.02-1.17) 1.09(1.02-1.16) 1.07(1.01-1.14) NS 
and CA 

H pylori and 2.29(1.71-3.05) 1.75(1.42-2.17) 1.57(1.29-1.90) 1.39(1.10-1.76) NS NS 
gastric CA 

cagA and gastric 2.87(1.95-4.22) 1.77(1.44-2.19) 1.68(1.34-2.09) 1.58(1.21-2.07) 1.58(1.03-2.43) NS 
CA 

Hpyloriand 1.77(1.45-2.14)" 1.59(1.30-1.95) 1.52(1.22-1.88) 1.42(1.12-1.79) 1.32(1.01-1.72) NS 
esophageal 
adeno-CA 

cagAand 2.41(1.61-3.61)" 2.02(1.32-3.10) 1.92(1.22-3.00) 1.75(1.03-2.96) NS NS 
esophageal 
adeno-CA 

HPV and tonsil CA 15.1(5.78-33.4) 14.63(4.71-45.4) 13.5(3.36-55.1) 11.3(1.55-77.2) NS NS 

HPV and cervical 8.08(5.04-10.8) 5.65(3.58-2.4) 5.65(3.0-14.78)6.65(1.98-22.42) NS NS 
CA 

HBV(HCV-)and 21.9(14.9-32.3) 10.5(5.75-19.5) 4.0(2.37-5.75)2.55(1.4-4.65) NS NS 
liver CA 

HCV(HBV-)and 20.3(12.2-33.7) 5.52(4.08-10.4) 6.4(3.55-11.4)6.23(2.51-14.9) 5.23(1.54-25.1) NS 
liver CA 

HBV + HCVand 51.2(27.0-139) 27.4(7.5-100.5)27.4(5.2-145.4)27.4(2.16-347.5) NS NS 
liver CA 

Abbreviations: OR, odds ratio; IGF, insulinlike growth factor; CRC, colorectal cancer; NS, not specified; BrCA, breast cancer; CA, cancer; SHBG, 
sex hormone-binding globulin; PrCA, prostate cancer; E2, estradiol; H pylori, Helicobacter pylori; HPV, human papillomavirus; HBV, hepatitis B 
virus; HCV, hepatitis C virus. 

" Effect inversed so that it is > 1.00. 



and 3 (30%) in the group with 95% prediction intervals 
excluding the null and 30 (88%), 26 (79%), 11 (33%), 
and 4 (12%), respectively, in those with 95% prediction in- 
tervals including the null. Thus, meta-analyses with 95% 
predictive intervals excluding the null were more likely to 
retain nominal statistical significance with various ceilings 
(e.g., P < 0.0001 for c = 20% for OR > 1.0). 

3.7. A working example of applying the credibility 
ceiling method 

To demonstrate the application of the method, we use the 
meta-analysis on IGF-2 and colorectal cancer [33] with 



synthesized ORs across three studies with a total sample size 
of 384 cases and 1,301 controls. The ORs (95% CI) in the 
three studies are 2.08 (0.85, 5.06), 2.09 (1.14, 3.82), and 
1.59 (0.67, 3.76). The summary effect is statistically signif- 
icant with an OR point estimate of 1.95 and 95% CI from 
1.26 to 3.00 according to random effects model. We now re- 
perform the meta-analysis by adjusting the variance of each 
study according to the credibility ceiling c using a range of 
different values for c. For example, for c = 12%, the 95% 
CIs of the three studies become (0.61, 7.03), (0.61, 7.17), 
and (0.67, 3.76) and the summary OR and 95% CI become 
1.82 (0.99, 3.35). For OR > 1.2, the summary effect estimate 
statistical significance is lost for a larger ceiling of 19% 
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(1.82; 95% CI: 0.98, 3.35; Supplemental Figure 4 at www. 
jclinepi.com, graph number 2). 

In the more general case, the code written in Stata to 
estimate the summary effect with different ceilings c is 
given by: 

forvalues i=l/'r(max)' { 

quietly metan logrevOR newse if compid= = 'i',nograph 
random eform 

replace effectrandom2=r(ES) if compid= = 'i' 
replace seesrandom2=r(selogES) if compid= = 'i' 
replace effectrandom_12=r(ci_low) if compid= = 'i' 
replace effectrandom_u2=r(ci_upp) if compid= = 'i' 
replace p_z2=r(p_z) if compid= = 'i' 
replace i_sq2=r(i_sq) if compid= = 'i' 
} 

if prob=normal(z) and newvar=(logrevOR-logx)/zcl)'^2 if 
prob < ceilingwhere prob = chance to observe a result on 
the opposite direction than the point estimate, logrevOR is 
the logarithm of the reversed OR and ceiling is the credi- 
bility ceiling we set for each study. 



4. Discussion 

Here, we have applied the credibility ceiling method to 
an array of meta-analyses on biomarkers and cancer risk 
to provide a critical approach to interpreting meta- 
analytic results in this field. Most of the meta-analyses that 
we examined lost their initial statistical significance for a 
credibility ceiling of 10% for OR >1.0. This ceiling is 
probably lenient, given the perplexity of the different types 
of bias an observational study may have. A value c = 10% 
merely states that we can reach even up to 90% certainty 
about the presence of a nonnull biomarker association just 
with a single study. Only seven biomarkers, all of them rep- 
resenting infectious agents maintained statistical signifi- 
cance for c = 20% for OR >1.0. This was supported 
also from the predictive intervals calculated for each 
meta-analysis. Most of the proposed cancer biomarkers of 
other noninfectious types may well be spurious. 

Meta-analyses of IGF/insulin system lost most statisti- 
cally significant associations with a ceiling of 10%. Meta- 
analyses examining the association between IGF-1 or 
lGFBP-3 and risk of prostate cancer |36| or C-peptide 
and breast cancer [37 1 lost their statistical significance even 
from a ceiling of 5%. One meta-analysis on IGF-1 and post- 
menopausal breast cancer and one on glucose and pancre- 
atic cancer had predictive intervals that did not include 
the null value. Some of these associations have been criti- 
cized for their magnitude as a survey has shown that au- 
thors of the primary studies on the association between 
IGF-I and prostate cancer supported the presence of strong 
evidence, whereas methodologists supported the presence 
of a null or a small effect |62|. 

Meta-analyses of inflammation biomarkers (CRP and 
colorectal [52] and other cancers [53]) also lost their 



statistical significance even with a ceiling of 5% for OR 
>1.0. This was reflected also in the predictive intervals. 
CRP appears to be an overrated biomarker for diverse out- 
comes, including both malignancies and cardiovascular 
outcomes, with no apparent causal relationship to any of 
them [6,7 1 . 

The fields of sex hormones, diet, and environment did 
not overall provide evidence of statistically significant asso- 
ciations. Only 1 of 13 and 1 of 31 had predictive intervals 
that did not include OR = 1. Practically, no study in the 
field of sex hormones and cancer risk survived a 10% ceil- 
ing. This is in agreement with a consortium-based analysis 
that also did not find any significant association between 
sex hormones and prostate cancer risk [38[. Nineteen 
percent of the meta-analyses on dietary associations had 
statistically significant findings, and of those, only two sur- 
vived the 10% ceiling for OR > 1.00 [48,51], both with a 
small number of synthesized studies and small number of 
cases. Large prospective cohort studies had little evidence 
to support these nutritional associations [63 [, and implau- 
sible results are common in published nutrition research 
[64,65 [. Finally, in the area of environmental biomarkers, 
only one meta-analysis regarding DNA adducts and cancer 
risk in current smokers had a statistically significant associ- 
ation, but it did not survive a 5% ceiling for OR> 1.0. 

The field of infectious agents had robust evidence for the 
presence of not only significant associations with cancer 
but also large effects, which is one of the traditional criteria 
of causality [66,67]. However, several meta-analyses lost 
statistical significance with a ceiling of only 10% for 
OR > 1.0 (e.g., H pylori and colorectal cancer, HPV and 
bladder/oral cancer, and Trichomonas vaginalis and cervi- 
cal cancer). This may reflect the phenomenon where when 
a strong risk factor is documented (H pylori and stomach 
cancer and HPV and cervical cancer), other associations 
are proposed also on the same pathogen but different cancer 
type or other pathogens for the same cancer type. It is un- 
clear whether the literature on these additional associations 
is reliable. For example, one may argue that some patho- 
gens may have pleiotropic effects on multiple cancer sites. 
Conversely, bias may lead to the creation of a spurious liter- 
ature where proposed associations "spill over" into other 
cancer types. 

Credibility ceilings may be applied for different OR 
values. We observed that when ceilings were assumed for 
OR > 1.2 rather than OR > 1.0, the loss in statistically sig- 
nificant signals was less prominent. This is not surprising, 
because for the same c value, a ceiling for the presence 
of an effect OR > 1.2 is a lesser constrain than the same 
ceiling for the presence of OR > 1 .0. With ceilings pertain- 
ing to OR > 1.2, meta-analyses that include studies finding 
mostly small effects may not be as much affected and such 
associations may withhold even substantial ceilings, as 
shown by several examples of noninfectious biomarkers. 
However, it is unclear whether such small effects are 
genuine or easily distinguishable from noise. Biomarkers 
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with OR < 1 .2 are unlikely to contribute much to risk of 
prognostic discrimination, but they can still create major 
debates about whether the estimated risks are real, 
let alone causal, even when large-scale evidence is accumu- 
lated [62]. 

In a previous evaluation, we had identified 24 meta- 
analyses that did not have hints of bias, including small- 
study effects, excess statistical significance, and large 
between-study heterogeneity 1 5 1 . These meta-analyses were 
more likely to retain statistical significance with the appli- 
cation of credibility ceilings than those that had at least one 
hint of bias. The difference was not very strong, however, 
suggesting that credibility ceilings offer independent infor- 
mation from what other tests of bias can offer. Moreover, 
we should acknowledge that no tests of bias can provide 
definite proof for (or exclude) bias and they may be sensi- 
tive to various assumptions [17,24,68,69]. Using the predic- 
tive interval approach, meta-analyses that did not include 
the null value in their predictive intervals were far more 
likely to retain statistical significance with increasing ceil- 
ings compared with those that included the null in their pre- 
dictive interval. 

Some additional limitations should be considered. First, 
the ceiling does not affect the point estimate of each study, 
so it is assumed that the point estimates are not biased. This 
is probably too conservative because in the presence of 
bias, it is expected that the point estimates would be in- 
flated. However, the method aims simply at decreasing 
the credence that can be placed in any single study, not 
to correct it for bias. As mentioned earlier, there are other 
more complex methods |9— 12, 14,15] that change both 
the maximum likelihood estimate and standard error of 
the studies, but solid knowledge on what corrections to 
apply is often lacking. We have avoided the subjectivity 
of choosing only certain ceiling values, and instead, we 
applied a considerable range of ceilings and interpreted 
the results accordingly. 

Second, the ceiling method inflates the variance to a 
greater extent in studies that have a point estimate away 
from the null. Studies with more extreme effects tend to 
obtain disproportionately greater variances, and as the ceil- 
ings become larger, the summary effects are more likely to 
get closer to the null. 

Finally, the ceiling method will reduce the power of a 
meta-analysis to detect an effect, and this reduction can 
be considerable. However, it creates a framework where 
repeated replication of an association becomes more impor- 
tant. Nominal statistical significance of an association in 
ceiling-adjusted calculations requires the conduct of many 
studies showing a consistent effect. Especially in the field 
of biomarkers, their cost utility depends on their true 
discriminating ability, so biomarkers with the potential to 
be used in large populations should be carefully adopted 
based on robust evidence in multiple studies. 

The application of this simple method with the afore- 
mentioned limitations may allow researchers to address 



in part the problems of bias in observational studies 
of biomarkers. Biomarkers are an appealing tool to use 
in clinical practice, and similar applications of the method 
in fields beyond cancer are warranted. Instead of trying 
to address the potential biases and confounding factors 
one by one (a formidable, if not impossible task), 
this method provides a skeptical approach to the observa- 
tional associations of biomarkers in total and may help 
avoiding the propagation of unjustified claims in this 
literature. 



Supplementary data 

Supplementary data related to this article can be found at 
http://dx.doi.Org/10.I016/j.jchnepi.20I4.09.004. 
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